E2E backups #4019

dbkr · 2018-10-08T14:02:53Z

Supersedes #3757 and #2731

Note that most of this has already been reviewed when it was in 2731 (unsure if it would be easier or more confusing to merge this branch over to that one and re-open that PR).

This is MSC1687: https://github.com/uhoreg/matrix-doc/blob/e2e_backup/proposals/1219-storing-megolm-keys-serverside.md

Sytests: matrix-org/sytest#503

uhoreg · 2018-10-10T19:53:58Z

I took a quick look at @dbkr's changes, and they look fine to me. It would be great to get a full review from a Synapse dev.

erikjohnston

Looks good! Just needs a little bit of sprucing up.

FTR: I've only really been reviewing the code, rather than trying to grok how E2E backups are meant to work as a whole.

For future reference, it'd also be good to avoid using @defer.inlineCallbacks in tests and instead use the test reactor infrastructure (which is also fairly nice). There are a number of advantages to this (Though personally the main one is that if you CTRL^C trial it sometimes completely wedges if using a real reactor)

erikjohnston · 2018-10-12T09:29:42Z

synapse/api/errors.py

+        return exception.error_dict()
+    else:
+        logger.error("Unknown exception type: %s", type(exception))
+        return {}


This appears unused?

mm, so it does

erikjohnston · 2018-10-12T09:30:04Z

synapse/handlers/e2e_room_keys.py

@@ -0,0 +1,283 @@
+# -*- coding: utf-8 -*-
+# Copyright 2017 New Vector Ltd


erikjohnston · 2018-10-12T09:31:44Z

synapse/handlers/e2e_room_keys.py

+        room, or a given session.
+        See EndToEndRoomKeyStore.get_e2e_room_keys for full details.
+
+        Returns:


Can you also add Args section, giving types (and description if not obvious)

(And the same for the other functions.)

erikjohnston · 2018-10-12T09:33:39Z

synapse/handlers/e2e_room_keys.py

+                    if e.code == 404:
+                        raise SynapseError(404, "Version '%s' not found" % (version,))
+                    else:
+                        raise e


Please just use raise rather than raise e as you end up losing a bunch of stack traces

ah, good to know

erikjohnston · 2018-10-12T09:34:26Z

synapse/handlers/e2e_room_keys.py

+        if E2eRoomKeysHandler._should_replace_room_key(current_room_key, room_key):
+            yield self.store.set_e2e_room_key(
+                user_id, version, room_id, session_id, room_key
+            )


Does this not want to be in the lock?

where by lock you mean try block? I don't think so as the try is just catching 404s from the get.

erikjohnston · 2018-10-12T09:44:15Z

synapse/rest/client/v2_alpha/room_keys.py

+        """
+
+        if version:
+            raise SynapseError(405, "Cannot POST to a specific version")


Can we please split this servlet into two: 1) which doesn't accept a version and only works for PUT and 2) which requires a version and only works with GET/DELETE.

This has the advantages of:

Its easier to grasp the API shapes without having to read the code

Don't have to re-implement all the validations/error handling

It does somewhat look like we've forgotten to check that we're actually given a version in GET/DELETE handlers for example

erikjohnston · 2018-10-12T09:45:20Z

synapse/rest/client/v2_alpha/room_keys.py

+            if e.code == 404:
+                e.errcode = Codes.NOT_FOUND
+                e.msg = "No backup found"
+                raise e


Can you make a new error rather than rewriting the existing one please? Otherwise we may end up with bizarre hybrid exceptions :)

TBH, I'm not even sure we want to bother rewriting anything here. If we do, we should probably do the same for the DELETE handler.

In fact, for consistency sake we should probably make the exceptions returned by the handler on unknown version be consistent, rather than the exception being specific to each REST endpoint.

erikjohnston · 2018-10-12T09:48:34Z

synapse/storage/e2e_room_keys.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import simplejson as json


We use just import json nowadays

erikjohnston · 2018-10-12T09:49:46Z

synapse/storage/e2e_room_keys.py

+            session_id(str): the session whose room_key we're setting
+            room_key(dict): the room_key being set
+        Raises:
+            StoreError if stuff goes wrong, probably


I don't think its worth having this comment :)

erikjohnston · 2018-10-12T09:56:16Z

synapse/rest/client/v2_alpha/room_keys.py

@@ -0,0 +1,361 @@
+# -*- coding: utf-8 -*-
+# Copyright 2017 New Vector Ltd


dbkr

Hopefully all addressed now - ptal

dbkr · 2018-10-12T10:04:57Z

synapse/api/errors.py

+        return exception.error_dict()
+    else:
+        logger.error("Unknown exception type: %s", type(exception))
+        return {}


mm, so it does

dbkr · 2018-10-12T10:15:30Z

synapse/handlers/e2e_room_keys.py

+        room, or a given session.
+        See EndToEndRoomKeyStore.get_e2e_room_keys for full details.
+
+        Returns:


dbkr · 2018-10-12T10:15:50Z

synapse/handlers/e2e_room_keys.py

+                    if e.code == 404:
+                        raise SynapseError(404, "Version '%s' not found" % (version,))
+                    else:
+                        raise e


ah, good to know

dbkr · 2018-10-12T10:19:48Z

synapse/handlers/e2e_room_keys.py

+        if E2eRoomKeysHandler._should_replace_room_key(current_room_key, room_key):
+            yield self.store.set_e2e_room_key(
+                user_id, version, room_id, session_id, room_key
+            )


where by lock you mean try block? I don't think so as the try is just catching 404s from the get.

dbkr · 2018-10-12T10:20:33Z

synapse/handlers/e2e_room_keys.py

+            # purely for legibility.
+
+            if room_key['is_verified'] and not current_room_key['is_verified']:
+                pass


dbkr · 2018-10-12T10:23:11Z

synapse/handlers/e2e_room_keys.py

+        """
+
+        results = yield self.store.get_e2e_room_keys_version_info(user_id, version)
+        defer.returnValue(results)


mm, I see what you mean

dbkr · 2018-10-12T10:35:27Z

synapse/rest/client/v2_alpha/room_keys.py

+        """
+
+        if version:
+            raise SynapseError(405, "Cannot POST to a specific version")


dbkr · 2018-10-12T10:47:47Z

synapse/rest/client/v2_alpha/room_keys.py

+            if e.code == 404:
+                e.errcode = Codes.NOT_FOUND
+                e.msg = "No backup found"
+                raise e


erikjohnston

Before this goes live we should probably move this functionality into a worker, as it sounds like this API is going to get hit a lot

richvdh · 2018-10-17T16:00:05Z

@dbkr please please please can we squash-merge things like this in future?

**Warning**: This release removes the example email notification templates from `res/templates` (they are now internal to the python package). This should only affect you if you (a) deploy your Synapse instance from a git checkout or a github snapshot URL, and (b) have email notifications enabled. If you have email notifications enabled, you should ensure that `email.template_dir` is either configured to point at a directory where you have installed customised templates, or leave it unset to use the default templates. The configuration parser will try to detect the situation where `email.template_dir` is incorrectly set to `res/templates` and do the right thing, but will warn about this. Features -------- - Ship the example email templates as part of the package ([\#4052](#4052)) - Add support for end-to-end key backup (MSC1687) ([\#4019](#4019)) Bugfixes -------- - Fix bug which made get_missing_events return too few events ([\#4045](#4045)) - Fix bug in event persistence logic which caused 'NoneType is not iterable' ([\#3995](#3995)) - Fix exception in background metrics collection ([\#3996](#3996)) - Fix exception handling in fetching remote profiles ([\#3997](#3997)) - Fix handling of rejected threepid invites ([\#3999](#3999)) - Workers now start on Python 3. ([\#4027](#4027)) - Synapse now starts on Python 3.7. ([\#4033](#4033)) Internal Changes ---------------- - Log exceptions in looping calls ([\#4008](#4008)) - Optimisation for serving federation requests ([\#4017](#4017)) - Add metric to count number of non-empty sync responses ([\#4022](#4022))

ara4n and others added 30 commits August 12, 2018 18:23

total WIP skeleton for /room_keys API

53ace90

interim WIP checkin; doesn't build yet

0bc4627

make it work and fix pep8

6b8c07a

document the API

cf1e200

implement /room_keys/version too (untested)

8ae64b2

make /room_keys/version work

69e51c7

blindly incorporate PR review - needs testing & fixing

0abb205

rename room_key_version table correctly, and fix opt args

cac0253

fix factoring out of _should_replace_room_key

ca0b052

add storage docstring; remove unused set_e2e_room_keys

8d14598

more docstring for the e2e_room_keys rest

9f500cb

add a tonne of docstring; make upload_room_keys properly assert version

9f0791b

add a tonne of docstring; make upload_room_keys properly assert version

14b3da6

fix typos

234611f

fix flakes

982edca

switch get_current_version_info back to being get_version_info

5e42c45

improve docstring

93d174b

don't needlessly return user_id

b5eee51

first cut at a UT

174be58

fix idiocies and so make tests pass

15d513f

linting

f6a3067

implement remaining tests and make them work

fe87890

flake8

edc427a

support DELETE /version with no args

72788cf

404 nicely if you try to interact with a missing current version

66a4ca1

use parse_string

54ac18e

missing import

f0cede5

missing import

4f7064f

allow auth_data to be any JSON instead of a string

8550a7e

allow session_data to be any JSON instead of just a string

42a394c

dbkr added 2 commits October 9, 2018 11:15

Add changelog

d34657e

Fix mergefail

b8d9e10

dbkr requested a review from a team October 9, 2018 17:49

dbkr mentioned this pull request Oct 9, 2018

E2e backups #3757

Closed

manuroe mentioned this pull request Oct 11, 2018

Implement E2E key backup element-hq/element-ios#2070

Closed

5 tasks

erikjohnston self-assigned this Oct 11, 2018

erikjohnston suggested changes Oct 12, 2018

View reviewed changes

erikjohnston assigned dbkr and unassigned erikjohnston Oct 12, 2018

dbkr added 4 commits October 12, 2018 11:26

PR feedback pt. 1

83e72bb

Split /room_keys/version into 2 servlets

86ef976

Don't mangle exceptions

bddfad2

Misc PR feedback bits

306361b

dbkr commented Oct 12, 2018

View reviewed changes

dbkr added 2 commits October 12, 2018 13:47

Linting soothes the savage PEP8 monster

8c0ff02

missed one

a45f2c3

dbkr assigned erikjohnston and unassigned dbkr Oct 12, 2018

erikjohnston approved these changes Oct 12, 2018

View reviewed changes

erikjohnston assigned dbkr and unassigned erikjohnston Oct 15, 2018

dbkr merged commit 03c1103 into develop Oct 15, 2018

richvdh mentioned this pull request Dec 13, 2018

Fix deleting e2e room keys on xenial #4295

Merged

uhoreg mentioned this pull request Oct 29, 2019

Proposal for storing megolm keys serverside matrix-org/matrix-spec-proposals#1219

Closed

DMRobertson deleted the dbkr/e2e_backups branch June 28, 2022 11:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E backups #4019

E2E backups #4019

dbkr commented Oct 8, 2018 •

edited

Loading

uhoreg commented Oct 10, 2018

erikjohnston left a comment

erikjohnston Oct 12, 2018

dbkr Oct 12, 2018

erikjohnston Oct 12, 2018

erikjohnston Oct 12, 2018

erikjohnston Oct 12, 2018

dbkr Oct 12, 2018

erikjohnston Oct 12, 2018

dbkr Oct 12, 2018

erikjohnston Oct 12, 2018

dbkr Oct 12, 2018

erikjohnston Oct 12, 2018

dbkr Oct 12, 2018

erikjohnston Oct 12, 2018

erikjohnston Oct 12, 2018

dbkr Oct 12, 2018

erikjohnston Oct 12, 2018

erikjohnston Oct 12, 2018

erikjohnston Oct 12, 2018

dbkr left a comment

dbkr Oct 12, 2018

dbkr Oct 12, 2018

dbkr Oct 12, 2018

dbkr Oct 12, 2018

dbkr Oct 12, 2018

dbkr Oct 12, 2018

dbkr Oct 12, 2018

dbkr Oct 12, 2018

erikjohnston left a comment

richvdh commented Oct 17, 2018

		@@ -0,0 +1,283 @@
		# -- coding: utf-8 --
		# Copyright 2017 New Vector Ltd

		@@ -0,0 +1,361 @@
		# -- coding: utf-8 --
		# Copyright 2017 New Vector Ltd

E2E backups #4019

E2E backups #4019

Conversation

dbkr commented Oct 8, 2018 • edited Loading

uhoreg commented Oct 10, 2018

erikjohnston left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbkr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikjohnston left a comment

Choose a reason for hiding this comment

richvdh commented Oct 17, 2018

dbkr commented Oct 8, 2018 •

edited

Loading